Systems for capturing, streaming, and/or playing back immersive content, such as to simulate an immersive virtual reality environment, e.g., a three-dimensional (3D) environment, are growing in importance as the hardware for such systems becomes more readily available and the cost of such hardware has been diminishing.
Head mounted devices for displaying immersive content can allow a user to turn his or her head and experience a corresponding change in the displayed scene. Such devices can allow users to adjust head position to view greater than about 180 degrees of a captured scene in at least one dimension, and can provide up to about 360 degree viewing in at least one dimension.
While some display devices allow for 360 degree viewing, at any given time human eyes are capable of perceiving a limited field of view corresponding to only a portion of the entire available viewing area. This field of view can be less than about 180 degrees in a horizontal dimension. Thus as the user turns their head, the playback device can adjust the displayed content to present the portion of the viewing scene corresponding to where the user is looking. For example, when the user is looking forward the display device presents a user with a sector of the scene centered about a forward direction, which can correspond to an angle of 0 degrees in a horizontal plane that is approximately parallel to the ground. When the user turns around to face backward, the display device presents the user with the sector of the scene centered about a backward direction, which can correspond to an angle of 180 degrees in the horizontal plane.
Some immersive systems capable of generating 360 degree viewing scenes typically combine content captured using multiple cameras. The cameras can capture stereoscopic content including image content corresponding to left and right eye views to allow for a stereoscopic 3D viewing effect. However, immersive experiences are possible without the use of stereoscopic content, e.g., with a user seeing the same image in both eyes and still obtaining a panoramic experience but without the sense of depth made possible by presenting different images to a user's left and right eyes.
In order to provide a wide field of view, cameras in immersive content capture systems can be fitted with wide angle lenses. These can include fisheye lenses or other ultra wide angle lenses that do not produce a rectilinear image. Fisheye lenses can produce strong visual distortion including significant barrel distortion, and create a wide panoramic or hemispherical image, which can make them well-suited for use in immersive content systems.
Spatial distortion, chromatic distortion, and/or other optical properties may vary from lens to lens and/or from camera to camera due factors including lens imperfections and differences between positioning of the lens relative to the sensor in the camera. This can make it difficult to reliably use images captured by cameras with fisheye lenses and to seam images together that are captured by adjacent cameras/lenses.
In the case of stereoscopic images, for example, where separate left and right eye images are captured and then presented during playback to produce a 3D effect to the viewer, distortions and differences between the cameras, including camera lenses, used to capture the left and right eye images may be particularly problematic and/or degrade the stereoscopic image quality if left unaddressed since the left and right eyes maybe presented with images suffering different distortions. Thus, while image distortion is a problem for panoramic implementations it is particularly problematic for stereoscopic systems where images captured by different cameras are displayed to different eyes.
In view of the above it should be appreciated that there is a need for detecting or measuring distortions introduced by cameras and camera lenses particularly in the case of wide angle, e.g., fish eye, lenses. There is also a need for need for methods and/or apparatus which could use distortion information to improve playback, e.g., when images captured by a camera are played back or used as a texture on a surface, e.g, of an environmental model. Methods and/or apparatus which can be used to address one or more of the above problems would be desirable.